AMOSSG/5-SN No. 22

29/3/05

AERODROME METEOROLOGICAL OBSERVING SYSTEMS STUDY GROUP (AMOSSG)

FIFTH MEETING

Montreal, 11 to 14 April 2005

Agenda Item	5:	Aerodrome observation requirements
5.2:	Capability of automatic observing equipment to meet aeronautical requirements and the use of automated equipment during operational hours

AUTOMATED VERSUS HUMAN ASSESSMENT OF VISIBILITY

(Presented by W. Maynard)

SUMMARY
This paper explains conflicting datasets on the comparative performance of humans and automation in assessing visibility. It concludes that the differences in the data sets are related to human limitations and that automated assessments of visibility should be preferred, wherever practicable, and especially when the application made of the assessment is of greater importance.

1.	INTRODUCTION

1.1	Three studies that include contrasting test results for the comparative abilities of humans and automated sensors to assess visibility are discussed.

1.2	An explanation is presented which proposes that the main reason for the divergent test results was attributable to the limitation of human and the use of "real world" visibility markers.

1.3	A clear implication is that only automated systems can provide a consistent and objective assessment of visibility. Furthermore, the variability in human assessments of visibility is such that automated methods of assessment should be preferred whenever practicable.

2.	DISCUSSION

2.1	Automated methods of assessing visibility have demonstrated a high level of reliability and consistency. Moreover, since the emergence of an international agreement on the definition of visibility for aviation purposes with amendment 72 to Annex 3, an objective basis is in place to establish consistency in automated assessment and reporting of visibility.

2.2	The World Meteorological Organization (WMO) Manual No. 8 — Guide to Meteorological Instruments and Methods of Observation — makes reference in Part I, Chapter 9, section 9.3.6 to The final report of The first WMO Intercomparison of Visibility Measurements which concluded that: "the best scatter meters showed little or no susceptibility to precipitation and provided estimates of MOR with standard deviation of about 10 per cent over a range of MOR from 100 m to 50 km." Note:— MOR means Meteorological Optical Range.

2.3	With respect to human observations, it is stated in WMO No. 8, Part I, Chapter 9, section 9.2.5 that;

Middleton (1952) found, from 1000 measurements, that the mean contrast ratio threshold for a group of 10 young airmen trained as meteorological observers was 0.033 with a range of 0.01 to more than 0.2. Sheppard (1983) has pointed out that when the Middleton data are plotted on a logarithmic scale they show good agreement with a Gaussian distribution. If the Middleton data represent normal observing conditions, we must expect daylight estimates of visibility to average about 14 per cent higher than MOR with a standard deviation of 20 per cent of MOR. These results are in excellent agreement with the results from the first WMO Intercomparison of Visibility Measurements (1990) where it was found that during daylight, the observers' estimates of visibility were about 15 per cent higher than instrumental measurements of MOR.

2.4	In contrast, a major Canadian study of automation made the opposite conclusion regarding the bias of human assessments of visibility. The Aviation Automated Weather Observation System (AWOS) Performance Evaluation (AAPE) concluded that automated reports of visibility are typically somewhat higher than comparable human observations. The AAPE study compared human and automated reports related to more than 70,000 METAR and 96,000 SPECI over a one year period at 8 sites representing a full range of Canadian climate.

2.5	The APPE final report stated;

—	"...when AWOS and a human observer do not agree, the propensity to report higher visibilities than the human is 2.5 times higher than it would be to report lower visibilities than the human."

and from its supplementary report;

—	"when there is disagreement, the AWOS more often reports higher, than lower visibilities compared to the observer."

further details of the results of this study are shown in the appendix to this paper.

2.6	It should be noted that if a pure Gaussian distribution is assumed in the human ability to assess visibility and that the standard deviation is of 20 percent then: 32 percent of human visibility assessments will vary from MOR by greater than 20 per cent; more than 4 per cent of human visibility assessment would vary by 40 percent or greater; and over 1 per cent would vary by a factor of half or more.

2.7	In the author's opinion an exactly Gaussian distribution is not likely a true characteristic of the uncertainty of human observations. One reason for this is that human will tend to observe to categories of significance whereas automated system will observe to all values without any bias. However, it can be said that any comparative test results between humans and automated methods of visibility assessment which demonstrate deviations on the scale of those described in 2.6 should not be considered as being significant deviations.

2.8	A number of theories had been suggested to account for the different conclusions regarding the human bias regarding visibility assessment. Among them; that the Canadian practice of observing prevailing visibility resulted in reduced visibility reports from humans, the reality that humans report towards categories, that the humans gave added weight to the existence of any lower visibility conditions in any sector or in the vicinity of the aerodrome or that human engaged in operational tasks have a natural safety bias to report lower values whenever in doubt.

2.9	Although all of the factors suggested in 2.8 are valid components they do not offer a satisfactory explanation.

2.10	It is assumed that both the Middleton and the WMO Intercomparison studies used, to the extent practicable, perfectly black, rectangular objects placed ideally against the horizon and used dedicated and trained personnel in obtaining test data. On the other hand the AAPE data set was based on the random availability of operational humans engaged in a variety of tasks using the available "real world" visibility markers which may had few of the ideal characteristics (for example, a white building or a clump of trees etc.).

2.11	In consideration of the above, it is proposed that the comparative WMO and AAPEG conclusions are actually measuring the limitations of the human observers in the absence of ideal visibility markers. As a result, few conclusion regarding the accuracy of automated visibility sensors can actually be drawn from comparisons with human observations.

2.12	Humans can see and identify ideal visibility markers from a greater distance than they can see and identify non-ideal markers; that is the basis of the difference in the results. Therefore, the consistency of human observations in low visibility is seriously limited by availability of suitable markers.

2.13	A further conclusion, based on this, is that the only practicable basis for establishing the accuracy of visibility sensors is other automated methods, in particular transmissometers over the range for which this is possible. It should be noted that the deviations between human and automated visibility assessments tend to decrease at high visibility values, so that a broad comparison may still have value at higher visibility values.

2.14	The use of automation wherever practicable and especially where the application of the reports have the greatest impact on efficiency and safety is supported by this explanation. This is also consistent with the current Annex 3 standards regarding the assessment of RVR in Appendix 3, section 4.3.2.1. Furthermore, any suggestion that automated assessment of visibility at major airports should not be permitted may be counter-productive.

2.15	There is one significant limitation to automation that still remains. It must be realized that even if the automated value is much more likely to be reliable and accurate that it may not necessarily be more representative for a given application depending upon sensor placement and the current spatial distribution of the weather. A network of suitably located sensors, where such is cost effective to do so, is suggested as a means of addressing this limitation.

3.	ACTION

3.1	Study group members are asked to:

a)	review and comment on the explanation for the different findings regarding the bias of human observers;

b)	comment on the conclusion that any deviations between comparative datasets of human and automated visibility assessments that fall within the parameters of a Gaussian distribution should not be considered significant to aviation; and

c)	comment on the conclusion that the characteristics and limitations of human observers that must be assumed as a consequence of the explanation set forth in this paper means that automated methods of visibility assessment should be preferred over human methods for aviation purposes and especially so when the use made of the data is of particular importance.

— — — — — — — —

AMOSSG/5-SN No. 22

Table 1: Results of comparison of human observations with closest matches from automated sensor within 15 minutes of the time of the human observation. Shaded area shows the cases where the automated assessment was of a higher value than the comparable human observation.

The Automated Visibility Sensor reported (Statute Miles)…

When The

Human

Staff

Reported

(Statute

Miles)	0	0.1	0.2 and 0.3	0.4	0.5	0.6, 0.7 and 0.8	0.9 to 1.0	1.2 to 1.6	1.7 to 2.1	2.2 to 2.5	3.0	3.5 and 4.0	5.0 to 9.0	>9.0
0	0	73	22	2	1	0	0	1	0	1	3	1	9	15
1/8	9	155	234	11	4	5	3	1	0	0	0	0	0	0
1/4	2	94	405	74	22	35	12	10	4	2	3	3	2	1
3/8	1	12	151	82	20	42	16	12	3	3	3	0	1	2
1/2	1	28	224	190	116	218	100	69	22	21	13	5	13	10
5/8 and 3/4	1	8	100	128	108	352	222	143	41	58	12	16	24	9
1	2	5	34	36	43	246	309	299	78	103	38	21	52	17
1 1/8, 1 1/4, 1 1/2	0	1	14	25	15	129	220	449	209	182	92	64	60	42
1 3/4 and 2	0	2	9	12	10	42	109	338	304	366	185	147	217	80
2 1/4 and

2 1/2

0	0	3	1	0	12	29	89	85	231	107	108	111	40
3	0	0	13	10	6	29	41	146	197	336	446	473	442	201
3 1/2

and

4	0	0	6	3	5	18	29	58	69	147	206	435	563	254
5.0 to 9.0	0	0	9	7	5	15	38	73	78	162	317	649	2771	2051
>9.0	3	0	13	10	8	36	36	77	69	95	200	367	3341	44923

— END —